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Abstract 

The state-of-the-art pooling strategies for perceptual im¬ 
age quality assessment (IQA) are based on the mean and the 
weighted mean. They are robust pooling strategies which 
usually provide a moderate to high performance for differ¬ 
ent IQAs. Recently, standard deviation (SD) pooling was 
also proposed. Although, this deviation pooling provides a 
very high performance for a few IQAs, its performance is 
lower than mean poolings for many other IQAs. In this pa¬ 
per, we propose to use the mean absolute deviation (MAD) 
and show that it is a more robust and accurate pooling strat¬ 
egy for a wider range of IQAs. In fact, MAD pooling has the 
advantages of both mean pooling and SD pooling. The joint 
computation and use of the MAD and SD pooling strategies 
is also considered in this paper. Experimental results pro¬ 
vide useful information on the choice of the proper deviation 
pooling strategy for different IQA models. 


1. Introduction 

Automatic image quality assessment (IQA) plays a sig¬ 
nificant role in many image processing applications. IQA 
is commonly used for monitoring, benchmarking, image 
restoration and parameter optimization mini. Full refer- 
ence IQAs, which fall within the scope of this paper, evalu¬ 
ate the perceptual quality of a distorted image with respect 
to its reference image. IQAs mimic the average quality pre¬ 
dictions of human observers. This is a non-trivial task be¬ 
cause images may suffer from various types and degrees of 
distortions. 

Among IQAs, the mean squared error (MSE) is widely 
used because of its simplicity. However, in many situations, 
it does not correlate with human perception of image fidelity 
and quality HU. A number of popular and/or high per- 
formance IQAs are SSIM (Ml, MS-SSIM UTl, VIF EH, 
VSNR 0, MAD El, FSIM EH, GS Q, GMSD EO), VSI 
ED, and others ID 

Usually, IQAs measure local similarity and produce a 
similarity score by a pooling strategy. This local qual¬ 
ity measurement can be performed on the image, differ¬ 


ent representations of the original image, or their combi¬ 
nation. For example, SSIM and MSSSIM use statistics of 
smoothed source and distorted images. FSIM 1^ uses 
a phase-derived map and another gradient-based map for 
quality assessment. GS Q is a contrast and structure vari¬ 
ant metric that utilizes specialized gradient magnitude and 
image contrast of the image. GMSD also utilizes gra¬ 
dient magnitude. Many of the available full reference IQAs 
follow this top-down architecture 0. While average pool¬ 
ing (Ill[T tI[231171 and average weighted pooling ISl fTSlITTl 
are widely used in the literature, GMSD uses standard de¬ 
viation pooling 1201 . For the purpose of this paper, we will 
take an overview of the FSIM 1231 and GMSD |20l indices. 

The feature similarity (FSIM) index l23l uses phase con¬ 
gruency (PC) 13 as its main feature, and an image gra¬ 
dient magnitude G as its secondary feature. Phase con¬ 
gruency similarity Spc and gradient magnitude similar¬ 
ity Sg are calculated and then combined as 5 'l(x) = 
[5'pc'(x)]^ [5 'g(x)]^, where x is an image pixel. For the pur¬ 
pose of pooling, first, the maximum PC of reference and 
distorted images PCm is computed. FSIM is then com¬ 
puted by the following mean pooling: 


FSIM = 


Ex (>gL(x)PC^(x)) 
Ex PCm{V) 


( 1 ) 


FSIM is among the leading indices in the literature; how¬ 
ever, the high computation cost of the phase congruency 
makes it an inefficient index. 

GMSD uses the Prewitt operator to calculate gradient 
magnitudes of reference and distorted images, Gn and Gp. 
From these, a gradient magnitude similarity (GMS) is cal¬ 
culated by: 


GMS(x) 


2Gp(x)Gx)(x) + c 
G'tc(x) -h Gf,(x) -h c' 


( 2 ) 


where c is a positive constant that supplies numerical sta¬ 
bility. The GMSD is then calculated by a deviation pooling 
strategy, which is the standard deviation of GMS values. 
GMSD provides high performance for different datasets and 
is very efficient. 
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The image gradients are sensitive to image distortions; 
different local structures in a distorted image suffer from 
different degrees of degradations 1^ . This is the motiva¬ 
tion that authors in |[20l used to explore the standard vari¬ 
ation of the gradient-based local similarity map for over¬ 
all image quality prediction. The standard deviation is the 
square root of the mean of the squares of the individual devi¬ 
ations. A problem with standard deviation is that the larger 
deviations are overemphasized in the process of squaring 
the deviations, since taking the square root is not a com¬ 
plete reversal. 

In this paper, the underestimated mean absolute devia¬ 
tion pooling is proposed for IQAs as it is more tolerant of 
the large deviations. We show that the mean absolute devi¬ 
ation is a faster and more reliable pooling than the standard 
deviation for different IQAs. Also for some IQAs, a combi¬ 
nation of these two pooling strategies results in indices that 
perform better. At the same time, the joint calculation of the 
standard deviation and the mean absolute deviation is still 
efficient. 

2. Deviation pooling strategies 

Deviation pooling for IQAs is rarely used in the litera¬ 
ture, except the standard deviation used in GMSD 1201 . De¬ 
viation is the variation of data values compared to a measure 
of central tendency (MCT) such as the mean, median, or 
mode. A deviation can be seen as the Minkowski distance 
of order p between vector x and a MCT: 

D(x,MCT)^ = (IZ “ MCT|'’) . (3) 

where, p > 1 indicates the type of deviation. For the pur¬ 
pose of this paper, it can be seen as: 

D(x, MCT)'’ = ^ |xi - MCT|'’) ^ (4) 

^ i = l 


be computed by: 


LS'^^ 



(LS(i)-LS^f. 


( 5 ) 


which is equal to equation 0 when p = 2. 

2.2. Mean absolute deviation (MAV) pooling 

Given the LS-^, the mean absolute deviation of 

an LS map is calculated by: 


= ^'^\LS{i) - LS^\. (6) 

^ i=l 

which is also equal to equation 0 when p = 1. In the 
experimental results section, the performance of the AiAV 
pooling based indices is evaluated. 

2.3. Double deviation {W) pooling 

We also found that a combination of the SV and MAV 
pooling strategies provides higher performance than either 
or for some IQAs. Furthermore, these two 

poolings can be computed at the same time using the fol¬ 
lowing formulas: 


MAV 


LS 


LS*®® = 


N 


1 


N 


-LS^\, 

(7) 

N 


Ea. 

(8) 

i=l 


N 


T.DI 

i=l 

(9) 


The joint computation of the and is like¬ 

wise computationally efficient. We call this combination the 
double deviation VV pooling strategy and compute it by: 


LS 


VV 


a + (1 - a) LS 


MAV 


( 10 ) 


Since above equation includes a MCT, it is different than 
the Minkowski pooling C6|, and the Minkowski metric m. 
In the following, the standard deviation pooling strategy in¬ 
troduced in GMSD is revisited. We then suggest using the 
mean absolute deviation pooling and show that these two 
pooling strategies can be jointly calculated. 

2.1. Standard deviation (SV) pooling 

Standard deviation is a simple and very common statisti¬ 
cal measure of the spread of scores within a set of data. Let 
LS-^ denote the mean of an arbitrary local similarity (LS) 
map computed by an IQ A model. Its standard deviation can 


where 0 < a < 1 adjusts the relative importance of the 
LS‘^^ and LS-^*^^ indices. We are able to combine these 
two pooling strategies because they share almost the same 
statistical characteristics and values. The usefulness of this 
combination is verified in the experiments. It should be 
noted that the median absolute deviation does not provide 
satisfactory performance, hence it is not evaluated in this 
paper. 

3. Experimental results 

In the experiments, four standard datasets are used. The 
LIVE dataset ifT^ contains 29 reference images and 779 
distorted images of five categories. The TID 2008 Co) 
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dataset contains 25 reference images and 1700 distorted im¬ 
ages. For each reference image, 17 types of distortions of 4 
degrees are available. CSIQ m is another dataset that con¬ 
sists of 30 reference images; each is distorted using six dif¬ 
ferent types of distortions at four to five levels of distortion. 
We also used the TID 2013 IH dataset, which contains 25 
reference images and 3000 distorted images. For each ref¬ 
erence image, 24 types of distortions of 5 degrees are avail¬ 
able. Also, three popular evaluation metrics were used in 
the experiments: the Spearman Rank order Correlation co¬ 
efficient (SRC), the Pearson linear Correlation Coefficient 
(PCC), and the Root Mean Square Error (RMSE). 

MSB, SSIM O, GS (H [a, FSIM (231 [21, GMSD 
|[20l [H, and VSI 1211 were used in the e^eriments us¬ 
ing two to three deviation pooling strategie^ Tables and 
[^provide a performance comparison between different in¬ 
dices using different pooling strategies. In reference to Fig. 

if W pooling provides higher performance than the oth¬ 
ers, its performance is added to Tables an dH Also, some 
of the state-of-the-art indices are added to the end of Tables 
[^and|^ In Figs. [T] an d@ a = 0 postulates to the MAV 
pooling, and a = 1 postulates to the SV pooling. In fact. 
Figs. and 1^ show SRC and PCC performance variations 
of different indices utilizing the VV pooling. 



Figure 1. The impact of the parameter a on the weighted average 
SRC (Table performance of the double deviation VV pooling 
strategy. 

From Tables[2and|^ it can be seen that MSB MAV per¬ 
forms better than the MSB, while the MSB SV shows the 
lowest performance among them. The original SSIM out¬ 
performs its deviation-based versions. The MAV pooling 
has higher performance than the SV pooling for SSIM. Bor 
SSIM, the deviation poolings show very low performance 
for some of the distortions, while the MAV pooling is still 
more robust than the SV pooling. 

Bor GS, the MAV pooling outperforms the others. SV 

^ Code: https://dl.dropboxusercontent.com/u/74505502/DeviationPoolings.m 



Figure 2. The impact of the parameter a on the weighted average 
PCC (Table performance of the double deviation VV pooling 
strategy. Note the MSB curve which is nonlinear. 

Table 2. Performance comparison of the different quality indices 
with and without deviation pooling strategies on TID 2013 dataset 


m. 


IQA 

TID2013 

SRC (distortions) 

SRC 

PCC 

avg 

min 

std 

MSE 

0.6394 

0.4785 

0.7951 

0.0766 

0.2355 

MSE SV 

0.6164 

0.5118 

0.7899 

0.0952 

0.2282 

MSE MAV 

0.6891 

0.7079 

0.7897 

0.0852 

0.2413 

SSIM tl 41 

0.7417 

0.7895 

0.8075 

0.3775 

0.1521 

SSIM SV 

0.7292 

0.7602 

0.7993 

0.0045 

0.2257 

SSIM MAV 

0.7463 

0.7806 

0.8041 

0.1471 

0.1989 

gtIiI 

0.7946 

0.8464 

0.8351 

0.3578 

0.1511 

GS SV 

0.7801 

0.8232 

0.8284 

0.3139 

0.1731 

GS MAV 

0.8081 

0.8613 

0.8332 

0.3344 

0.1584 

ESIM 

0.8015 

0.8589 

0.8219 

0.2748 

0.1662 

ESIM SV (20] 

0.8077 

0.8602 

0.8263 

0.2126 

0.1940 

ESIM MAV 

0.8051 

0.8547 

0.8327 

0.2691 

0.1706 

ESIMI2D (a = 0.5) 

0.8118 

0.8614 

0.8293 

0.2304 

0.1852 

GMS M 

0.7884 

0.8395 

0.8370 

0.3700 

0.1500 

GMSD [20] 

0.8044 

0.8590 

0.8300 

0.2948 

0.1815 

GMS MAV 

0.8084 

0.8608 

0.8369 

0.3450 

0.1597 

GMS VV {a = 0.5) 

0.8111 

0.8658 

0.8332 

0.3132 

0.1729 

VSI r2ii 

0.8965 

0.9000 

0.8514 

0.1713 

0.1787 

VSI SV 

0.8556 

0.8651 

0.8633 

0.3935 

0.1274 

VSI MAV 

0.8853 

0.8965 

0.8675 

0.4564 

0.1193 

WSIVV (a = 0.1) 

0.8830 

0.8951 

0.8680 

0.4542 

0.1183 

MSSSIM [17J 

0.7859 

0.8329 

0.8109 

0.4099 

0.1506 

VIEflll 

0.6769 

0.7720 

0.8267 

0.3099 

0.1464 

IWSSIM [15] 

0.7779 

0.8319 

0.7978 

0.3717 

0.1601 

MAD [4] 

0.7807 

0.8267 

0.7556 

0.0575 

0.2644 


pooling for GS does not provide a high performance be¬ 
cause GS uses image contrast, and we had already observed 
that SV pooling is not a good choice for MSE. While BSIM 
SV shows overall higher performance than BSIM MAV, 
its performance for some distortion types is low. Bor BSIM, 
W pooling with a = 0.5 provides higher overall perfor¬ 
mance than SV pooling. At the same time, W pooling 
shows better quality prediction on distortion types than SV 
pooling. The overall performance of the GMS MAV and 
GMS SV indices is competitive; however, GMS MAV 
shows better quality prediction on the distortion types. The 
overall performance of GMS W and its performance for 
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(a) original image 
taken from TID 2008 llOl 


(b) subjective score = 7.1290 
SV: GS = 0.0030, GMS = 0.0225 
MAV: GS = 0.0013, GMS = 0.0075 


(c) subjective score = 5.0645 
SV: GS = 0.0027, GMS = 0.0127 
MAV: GS = 0.0020, GMS = 0.0111 


Figure 3. A comparison of ST> and MAV for contrast distortion type, (a) original image, (b)-(c) contrast distortion images of (a). Note 
that higher subjective scores and lower GS/GMS scores indicate higher quality. Clearly, GS/GMS MAV provides better judgment than 
GS/GMS SV. 


Table 1. Performance comparison of the different quality indices with and without deviation pooling strategies on LIVE C3. CSIQ p4( and 
TID 2008 fT^ datasets. 


IQA 

LIVE (779 images) 

CSIQ (886 images) 

TID2008 (1700 images) 

Weighted avg 

SRC (distortions) 

SRC 

PCC 

RMSE 

SRC 

PCC 

RMSE 

SRC 

PCC 

RMSE 

SRC 

PCC 

avg 

min 

std 

MSE 

0.8756 

0.8723 

13.3597 

0.8058 

0.7512 

0.1733 

0.5531 

0.5734 

1.0994 

0.6943 

0.6894 

0.8473 

0.5815 

0.1078 

MSE SV (20] 

0.8771 

0.5707 

N/A 

0.8344 

0.6448 

0.2007 

0.5801 

0.5593 

1.1124 

0.7158 

0.5845 

0.8290 

0.2624 

0.1658 

MSE MAV 

0.8746 

0.8716 

13.3944 

0.8239 

0.5594 

0.2176 

0.6712 

0.6473 

1.0229 

0.7585 

0.6761 

0.8422 

0.5045 

0.1300 

SSIM[l4] 

SSIM SV (2^ 

0.9479 

0.9449 

8.9455 

0.8756 

0.8613 

0.1334 

0.7749 

0.7732 

0.8511 

0.8415 

0.8361 

0.8644 

0.5246 

0.1069 

0.9174 

0.9032 

11.7261 

0.8169 

0.8094 

0.1542 

0.7560 

0.7374 

0.9063 

0.8094 

0.7948 

0.8154 

0.0083 

0.2209 

SSIM MAV 

0.9166 

0.9017 

11.8156 

0.8388 

0.8316 

0.1458 

0.7775 

0.7619 

0.8691 

0.8258 

0.8126 

0.8255 

0.2059 

0.1836 

gsTt] 

0.9561 

0.9512 

8.4327 

0.9108 

0.8964 

0.1164 

0.8504 

0.8422 

0.7235 

0.8908 

0.8817 

0.8915 

0.6691 

0.0923 

GS SV 

0.9464 

0.9409 

9.2559 

0.9308 

0.9241 

0.1003 

0.8466 

0.8307 

0.7470 

0.8919 

0.8808 

0.8872 

0.6146 

0.1108 

GS MAV 

0.9538 

0.9486 

8.6494 

0.9295 

0.9198 

0.1030 

0.8823 

0.8720 

0.6568 

0.9113 

0.9023 

0.8899 

0.6523 

0.1025 

ESIM 

ESIM SV (20j 

0.9634 

0.9597 

7.6780 

0.9242 

0.9120 

0.1077 

0.8805 

0.8738 

0.6525 

0.9112 

0.9038 

0.8881 

0.6481 

0.0934 

0.9602 

0.9579 

7.8442 

0.9566 

0.9534 

0.0792 

0.8914 

0.8762 

0.6467 

0.9245 

0.9154 

0.8843 

0.4412 

0.1268 

ESIM MAV 

0.9609 

0.9580 

7.8370 

0.9525 

0.9460 

0.0851 

0.8783 

0.8634 

0.6770 

0.9170 

0.9071 

0.8916 

0.6195 

0.1043 

ESIMDD (a = 0.5) 

0.9611 

0.9584 

7.8003 

0.9555 

0.9507 

0.0814 

0.8935 

0.8775 

0.6436 

0.9255 

0.9155 

0.8882 

0.5066 

0.1164 

GMS A4 

0.9595 

0.9556 

8.0489 

0.9290 

0.9127 

0.1073 

0.8477 

0.8366 

0.7351 

0.8950 

0.8842 

0.8924 

0.6301 

0.0962 

GMSD (20) 

0.9603 

0.9603 

7.6214 

0.9570 

0.9541 

0.0786 

0.8907 

0.8788 

0.6404 

0.9243 

0.9175 

0.8849 

0.4659 

0.1246 

GMS MAV 

0.9627 

0.9618 

7.4802 

0.9532 

0.9457 

0.0853 

0.8837 

0.8711 

0.6589 

0.9203 

0.9118 

0.8935 

0.6224 

0.1034 

GMS VV (a = 0.5) 

0.9619 

0.9614 

7.5149 

0.9559 

0.9509 

0.0813 

0.8961 

0.8830 

0.6300 

0.9271 

0.9190 

0.8889 

0.5204 

0.1160 

VSI 1211 

0.9524 

0.9482 

8.6817 

0.9423 

0.9279 

0.0979 

0.8979 

0.8762 

0.6466 

0.9222 

0.9065 

0.8987 

0.6295 

0.1036 

VSI SV |2Q| 

0.9546 

0.9519 

8.3757 

0.9569 

0.9532 

0.0801 

0.8775 

0.8585 

0.6881 

0.9163 

0.9048 

0.8957 

0.5393 

0.1080 

WSl MAV 

0.9577 

0.9540 

8.1952 

0.9546 

0.9449 

0.0859 

0.9048 

0.8866 

0.6207 

0.9302 

0.9175 

0.9027 

0.6312 

0.0949 

WSIVV (a = 0.1) 

0.9575 

0.9539 

8.1958 

0.9553 

0.9462 

0.0849 

0.9048 

0.8872 

0.6193 

0.9303 

0.9182 

0.9029 

0.6275 

0.0948 

MSSSIM [17] 

0.9513 

0.9489 

8.6188 

0.9133 

0.8991 

0.1149 

0.8542 

0.8451 

0.7173 

0.8922 

0.8834 

0.8796 

0.6381 

0.0993 

VIEflll 

0.9636 

0.9604 

7.6137 

0.9195 

0.9277 

0.0980 

0.7491 

0.8084 

0.7899 

0.8436 

0.8750 

0.8949 

0.5102 

0.0987 

IWSSIM [T5] 

0.9567 

0.9522 

8.3472 

0.9213 

0.9144 

0.1063 

0.8559 

0.8579 

0.6895 

0.8965 

0.8946 

0.8708 

0.6301 

0.1063 

MAD (4) 

0.9669 

0.9675 

6.9073 

0.9466 

0.9500 

0.0820 

0.8340 

0.8290 

0.7505 

0.8944 

0.8929 

0.8387 

0.0650 

0.2152 


distortion types are simultaneously higher than GMS SV. 

VSI is a high performance similarity index that was re¬ 
cently proposed in 1^ . Using MAV pooling, its per¬ 
formance improved considerably on the first three datasets 
for all of the measures used in this paper. For the TID 
2013 dataset, however, its overall performance decreased 
by 1.2493% for SRC and by 0.3889% for PCC metrics. In 
turn, it has 1.8910% better average prediction on distortion 
types. Also, the minimum quality prediction of VSI im¬ 
proved from 0.1713 to the 0.4564. These advantages show 
that MAV pooling is a good choice for VSI. SV pooling 
shows the lowest performance for VSI. 

Overall, MAV pooling is more robust than SV pool¬ 
ing, especially for assessment of individual distortion types. 
The low min SRC values for SV pooling in Table show 
its unreliability in comparison to MAV pooling. In gen¬ 


eral, higher orders of p in equation 0 result in a worst and 
unstable assessment for distortion types. In other words, the 
std value in the last column of the Table increases by in¬ 
creasing the p value. It is worth noting that this fact may 
not always be true. 

Fig. [^shows an example in which SV pooling fails in as¬ 
sessment, while MAV pooling provides a true assessment 
for both the GS and GMS indices. Fig. shows the run time 
of the three pooling strategies used in this paper. Our exper¬ 
iments were performed on a Core i7 3.4 GHz CPU with 16 
GB of RAM running on MATLAB 2013b and Windows 7. 
MAV is the second fastest after the mean, while the joint 
calculation of SV and MAV is still efficient. Therefore, 
GMS MAV is even faster than the highly efficient GMSD. 
GMS W is slightly slower than GMSD; however, its im¬ 
proved performance over GMS SV is noticeable. 
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Figure 4. Run time versus the local similarity (LS) size of the mean 
pooling and three deviation pooling strategies used in this paper. 

4. Conclusion 

Deviation pooling strategies for full reference image 
quality assessment were analyzed. The mean absolute devi¬ 
ation (MAD) pooling and the standard deviation (SD) pool¬ 
ing strategies were compared on the basis of their effectiv- 
ity, robustness and efficiency. The computation of MAD is 
faster than SD, and this may be of high interest for design¬ 
ing more efficient indices. While none of them fully out¬ 
performed the others, MAD pooling shows a clear advan¬ 
tage of robustness over SD pooling. Furthermore, for some 
of the image quality assessment models, a combination of 
these two pooling strategies results in better performing in¬ 
dices. Considering the experimental results, we highly rec¬ 
ommend the use of MAD pooling for different image qual¬ 
ity assessment purposes. 
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